A Novel Generalized Value Iteration Scheme For Partially-Unknown Continuous-Time Linear Systems

In this paper, a novel generalized value iteration technique is presented for solving online the discounted linear quadratic (LQ) optimal control problems for continuous-time (CT) linear systems with an unknown system matrix A. In the proposed method, the discounted value function is considered, which is a general setting in reinforcement learning (RL) frameworks, but not fully considered in RL for CT dynamical systems. Moreover, stepwise-varying learning rate ηi is introduced for the fast and safe convergence. For the stability and monotone convergence to the true optimal solution, it is mathematically proven that if stepwise-varying learning rate ηi lies in some specified ranges, the proposed algorithm guarantees the so-called Hurwitz property concerning the stability of the closed loop system, and, in addition, converges to the discounted LQ optimal solution. These proofs also give the stability and monotone convergence conditions for the existing value iteration method as a special case since the proposed method is more general than the existing ones.